{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hyperparameter_tuning.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vhyPjzWq_Xb1",
        "colab_type": "text"
      },
      "source": [
        "## Hyperparameter Tuning Design Pattern\n",
        "\n",
        "In Hyperparameter Tuning, the training loop is itself inserted into an optimization method to find the optimal set of model hyperparameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NikTzWZr_cxP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import datetime\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import time\n",
        "\n",
        "from tensorflow import keras\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import accuracy_score, f1_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PFwgTDEpRhRU",
        "colab_type": "text"
      },
      "source": [
        "### Grid search in Scikit-learn\n",
        "\n",
        "Here we'll look at how to implement hyperparameter tuning with the grid search algorithm, using Scikit-learn's built-in `GridSearchCV`. We'll do this by training a random forest model on the UCI mushroom dataset, which predicts whether a mushroom is edible or poisonous."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T818NKZybNtz",
        "colab_type": "code",
        "outputId": "c8dcde65-0c15-432c-c1cf-2cfe5d156e48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# First, download the data\n",
        "# We've made it publicly available in Google Cloud Storage\n",
        "!gsutil cp gs://ml-design-patterns/mushrooms.csv ."
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Copying gs://ml-design-patterns/mushrooms.csv...\n",
            "/ [1 files][365.2 KiB/365.2 KiB]                                                \n",
            "Operation completed over 1 objects/365.2 KiB.                                    \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7cVhvWToGO5v",
        "colab_type": "code",
        "outputId": "2b9cb359-e6d8-45ee-c300-bcddd1e401a9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "mushroom_data = pd.read_csv('mushrooms.csv')\n",
        "mushroom_data.head()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>class</th>\n",
              "      <th>cap-shape</th>\n",
              "      <th>cap-surface</th>\n",
              "      <th>cap-color</th>\n",
              "      <th>bruises</th>\n",
              "      <th>odor</th>\n",
              "      <th>gill-attachment</th>\n",
              "      <th>gill-spacing</th>\n",
              "      <th>gill-size</th>\n",
              "      <th>gill-color</th>\n",
              "      <th>stalk-shape</th>\n",
              "      <th>stalk-root</th>\n",
              "      <th>stalk-surface-above-ring</th>\n",
              "      <th>stalk-surface-below-ring</th>\n",
              "      <th>stalk-color-above-ring</th>\n",
              "      <th>stalk-color-below-ring</th>\n",
              "      <th>veil-type</th>\n",
              "      <th>veil-color</th>\n",
              "      <th>ring-number</th>\n",
              "      <th>ring-type</th>\n",
              "      <th>spore-print-color</th>\n",
              "      <th>population</th>\n",
              "      <th>habitat</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>p</td>\n",
              "      <td>x</td>\n",
              "      <td>s</td>\n",
              "      <td>n</td>\n",
              "      <td>t</td>\n",
              "      <td>p</td>\n",
              "      <td>f</td>\n",
              "      <td>c</td>\n",
              "      <td>n</td>\n",
              "      <td>k</td>\n",
              "      <td>e</td>\n",
              "      <td>e</td>\n",
              "      <td>s</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>w</td>\n",
              "      <td>p</td>\n",
              "      <td>w</td>\n",
              "      <td>o</td>\n",
              "      <td>p</td>\n",
              "      <td>k</td>\n",
              "      <td>s</td>\n",
              "      <td>u</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>e</td>\n",
              "      <td>x</td>\n",
              "      <td>s</td>\n",
              "      <td>y</td>\n",
              "      <td>t</td>\n",
              "      <td>a</td>\n",
              "      <td>f</td>\n",
              "      <td>c</td>\n",
              "      <td>b</td>\n",
              "      <td>k</td>\n",
              "      <td>e</td>\n",
              "      <td>c</td>\n",
              "      <td>s</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>w</td>\n",
              "      <td>p</td>\n",
              "      <td>w</td>\n",
              "      <td>o</td>\n",
              "      <td>p</td>\n",
              "      <td>n</td>\n",
              "      <td>n</td>\n",
              "      <td>g</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>e</td>\n",
              "      <td>b</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>t</td>\n",
              "      <td>l</td>\n",
              "      <td>f</td>\n",
              "      <td>c</td>\n",
              "      <td>b</td>\n",
              "      <td>n</td>\n",
              "      <td>e</td>\n",
              "      <td>c</td>\n",
              "      <td>s</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>w</td>\n",
              "      <td>p</td>\n",
              "      <td>w</td>\n",
              "      <td>o</td>\n",
              "      <td>p</td>\n",
              "      <td>n</td>\n",
              "      <td>n</td>\n",
              "      <td>m</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>p</td>\n",
              "      <td>x</td>\n",
              "      <td>y</td>\n",
              "      <td>w</td>\n",
              "      <td>t</td>\n",
              "      <td>p</td>\n",
              "      <td>f</td>\n",
              "      <td>c</td>\n",
              "      <td>n</td>\n",
              "      <td>n</td>\n",
              "      <td>e</td>\n",
              "      <td>e</td>\n",
              "      <td>s</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>w</td>\n",
              "      <td>p</td>\n",
              "      <td>w</td>\n",
              "      <td>o</td>\n",
              "      <td>p</td>\n",
              "      <td>k</td>\n",
              "      <td>s</td>\n",
              "      <td>u</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>e</td>\n",
              "      <td>x</td>\n",
              "      <td>s</td>\n",
              "      <td>g</td>\n",
              "      <td>f</td>\n",
              "      <td>n</td>\n",
              "      <td>f</td>\n",
              "      <td>w</td>\n",
              "      <td>b</td>\n",
              "      <td>k</td>\n",
              "      <td>t</td>\n",
              "      <td>e</td>\n",
              "      <td>s</td>\n",
              "      <td>s</td>\n",
              "      <td>w</td>\n",
              "      <td>w</td>\n",
              "      <td>p</td>\n",
              "      <td>w</td>\n",
              "      <td>o</td>\n",
              "      <td>e</td>\n",
              "      <td>n</td>\n",
              "      <td>a</td>\n",
              "      <td>g</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  class cap-shape cap-surface  ... spore-print-color population habitat\n",
              "0     p         x           s  ...                 k          s       u\n",
              "1     e         x           s  ...                 n          n       g\n",
              "2     e         b           s  ...                 n          n       m\n",
              "3     p         x           y  ...                 k          s       u\n",
              "4     e         x           s  ...                 n          a       g\n",
              "\n",
              "[5 rows x 23 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZITDo6upbV-C",
        "colab_type": "text"
      },
      "source": [
        "To keep things simple, we'll first convert the label column to numeric and then \n",
        "use `pd.get_dummies()` to covert the data to numeric. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_W5aElS8H5g5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 1 = edible, 0 = poisonous\n",
        "mushroom_data.loc[mushroom_data['class'] == 'p', 'class'] = 0\n",
        "mushroom_data.loc[mushroom_data['class'] == 'e', 'class'] = 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oP6HGdFZHemb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "labels = mushroom_data.pop('class')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTjqVF79G67V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dummy_data = pd.get_dummies(mushroom_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4VWcj1hHrZ6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Split the data\n",
        "train_size = int(len(mushroom_data) * .8)\n",
        "\n",
        "train_data = dummy_data[:train_size]\n",
        "test_data = dummy_data[train_size:]\n",
        "\n",
        "train_labels = labels[:train_size].astype(int)\n",
        "test_labels = labels[train_size:].astype(int)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9RUF2atVnEBc",
        "colab_type": "text"
      },
      "source": [
        "Next, we'll build our Scikit-learn model and define the hyperparameters we want to optimize using grid serach."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JUvky1eWFprB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = RandomForestClassifier()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kJb0SaDJ7le",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "grid_vals = {\n",
        "  'max_depth': [5, 10, 100],\n",
        "  'n_estimators': [100, 150, 200]\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QRIMtP_QKS6W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "grid_search = GridSearchCV(model, param_grid=grid_vals, scoring='accuracy')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "THP9fU5RNm15",
        "colab_type": "code",
        "outputId": "3898ba63-7fb9-402e-ff06-d6a4b381cc5f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "# Train the model while running hyperparameter trials\n",
        "grid_search.fit(train_data.values, train_labels.values)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=None, error_score=nan,\n",
              "             estimator=RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
              "                                              class_weight=None,\n",
              "                                              criterion='gini', max_depth=None,\n",
              "                                              max_features='auto',\n",
              "                                              max_leaf_nodes=None,\n",
              "                                              max_samples=None,\n",
              "                                              min_impurity_decrease=0.0,\n",
              "                                              min_impurity_split=None,\n",
              "                                              min_samples_leaf=1,\n",
              "                                              min_samples_split=2,\n",
              "                                              min_weight_fraction_leaf=0.0,\n",
              "                                              n_estimators=100, n_jobs=None,\n",
              "                                              oob_score=False,\n",
              "                                              random_state=None, verbose=0,\n",
              "                                              warm_start=False),\n",
              "             iid='deprecated', n_jobs=None,\n",
              "             param_grid={'max_depth': [5, 10, 100],\n",
              "                         'n_estimators': [100, 150, 200]},\n",
              "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
              "             scoring='accuracy', verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v0T3wwGennxK",
        "colab_type": "text"
      },
      "source": [
        "Let's see which hyperparameters resulted in the best accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IOXd0OoInsp0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fbfe666a-5766-4eb4-9476-c0276d17c369"
      },
      "source": [
        "grid_search.best_params_"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'max_depth': 10, 'n_estimators': 150}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OEhnb4oOnvgd",
        "colab_type": "text"
      },
      "source": [
        "Finally, we can generate some test predictions on our model and evaluate its accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cn9esE0_NniN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "grid_predict = grid_search.predict(test_data.values)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pI1ZBLmZNuhE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "grid_acc = accuracy_score(test_labels.values, grid_predict)\n",
        "grid_f = f1_score(test_labels.values, grid_predict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yuufp8PmOBWS",
        "colab_type": "code",
        "outputId": "c950c312-6e8a-402b-dd4d-f9a3fa5480cb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "print('Accuracy: ', grid_acc)\n",
        "print('F1-Score: ', grid_f)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.9950769230769231\n",
            "F1-Score:  0.9921722113502935\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X0DgrKkDRoPc",
        "colab_type": "text"
      },
      "source": [
        "### Hyperparameter tuning with `keras-tuner`\n",
        "\n",
        "To show how this works we'll train a model on the MNIST handwritten digit dataset, which is available directly in Keras. For more details, see this [Keras tuner guide](https://www.tensorflow.org/tutorials/keras/keras_tuner)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zg0ZehhZXjYD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "acefb89c-cc6f-4892-c614-08efaffd31d6"
      },
      "source": [
        "!pip install keras-tuner --quiet"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K     |██████                          | 10kB 16.6MB/s eta 0:00:01\r\u001b[K     |████████████                    | 20kB 1.8MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 30kB 2.3MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 40kB 1.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 51kB 2.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 61kB 1.8MB/s \n",
            "\u001b[?25h  Building wheel for keras-tuner (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for terminaltables (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_e7tpciTxXby",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import kerastuner as kt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9D4yIgYSXEZs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "b11af6d7-ae33-46cf-8c34-603f2b2a17fd"
      },
      "source": [
        "# Get the mnist data\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "srU9F3HLcn6O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_model(hp):\n",
        "    model = keras.Sequential([\n",
        "      keras.layers.Flatten(input_shape=(28, 28)),\n",
        "      keras.layers.Dense(hp.Int('first_hidden', 128, 256, step=32), activation='relu'),\n",
        "      keras.layers.Dense(hp.Int('second_hidden', 16, 128, step=32), activation='relu'),\n",
        "      keras.layers.Dense(10, activation='softmax')\n",
        "    ])\n",
        "\n",
        "    model.compile(\n",
        "      optimizer=tf.keras.optimizers.Adam(\n",
        "        hp.Float('learning_rate', .005, .01, sampling='log')),\n",
        "      loss='sparse_categorical_crossentropy', \n",
        "      metrics=['accuracy'])\n",
        "  \n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_yl-LDjXp3n",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "17025f06-a005-4555-b040-b2e2f10f389a"
      },
      "source": [
        "tuner = kt.BayesianOptimization(\n",
        "    build_model,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=30\n",
        ")"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Reloading Oracle from existing project ./untitled_project/oracle.json\n",
            "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter\n",
            "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1\n",
            "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2\n",
            "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay\n",
            "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
            "WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.\n",
            "INFO:tensorflow:Reloading Tuner from ./untitled_project/tuner0.json\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6zGqVhpLm9di",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner.search(x_train, y_train, validation_split=0.1, epochs=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNBFLJ6-02Tk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "best_hps = tuner.get_best_hyperparameters(num_trials = 1)[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fT5Ijj-TRuu_",
        "colab_type": "text"
      },
      "source": [
        "### Hyperparameter tuning on Cloud AI Platform\n",
        "\n",
        "In this section we'll show you how to scale your hyperparameter optimization by running it on Google Cloud's AI Platform. You'll need a Cloud account with AI Platform Training enabled to run this section.\n",
        "\n",
        "We'll be using PyTorch to build a regression model in this section. To train the model we'll be the BigQuery natality dataset. We've made a subset of this data available in a public Cloud Storage bucket, which we'll download from within the training job."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3HKkQk9rABYd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4s44baKLcrwl",
        "colab_type": "text"
      },
      "source": [
        "In the cells below, replcae `your-project-id` with the ID of your Cloud project, and `your-gcs-bucket` with the name of your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0bL6C3hYDeYg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cc469a4b-e728-47aa-ea54-49c9595cfc7a"
      },
      "source": [
        "!gcloud config set project your-project-id"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Updated property [core/project].\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVgtxOraEMdJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BUCKET_URL = 'gs://your-gcs-bucket'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7g1ybOsbboIv",
        "colab_type": "text"
      },
      "source": [
        "To run this on AI Platform, we'll need to package up our model code in Python's package format, which includes an empty `__init__.py` file and a `setup.py` to install dependencies (in this case PyTorch, Scikit-learn, and Pandas)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QD4t0GqrDOPe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "93bf3395-9c34-49f9-f10e-9eca0bf59f7d"
      },
      "source": [
        "!mkdir trainer\n",
        "!touch trainer/__init__.py"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mkdir: cannot create directory ‘trainer’: File exists\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GMsEoTxDFuSu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9013576d-f7c8-4cf1-b12b-5e01ade6d4f3"
      },
      "source": [
        "%%writefile setup.py\n",
        "from setuptools import find_packages\n",
        "from setuptools import setup\n",
        "\n",
        "REQUIRED_PACKAGES = ['torch>=1.5', 'scikit-learn>=0.20', 'pandas>=1.0']\n",
        "\n",
        "setup(\n",
        "    name='trainer',\n",
        "    version='0.1',\n",
        "    install_requires=REQUIRED_PACKAGES,\n",
        "    packages=find_packages(),\n",
        "    include_package_data=True,\n",
        "    description='My training application package.'\n",
        ")"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting setup.py\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8spoFJascZBm",
        "colab_type": "text"
      },
      "source": [
        "Below, we're copying our model training code to a `model.py` file in our trainer package directory. This code runs training and after training completes, reports the model's final loss to Cloud HyperTune."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SRcyqCs7MnU4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "0aa5f5bd-ae2f-4382-a7bc-67fc89d09404"
      },
      "source": [
        "%%writefile trainer/model.py\n",
        "import argparse\n",
        "import hypertune\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.preprocessing import normalize\n",
        "\n",
        "def get_args():\n",
        "    \"\"\"Argument parser.\n",
        "    Returns:\n",
        "        Dictionary of arguments.\n",
        "    \"\"\"\n",
        "    parser = argparse.ArgumentParser(description='PyTorch MNIST')\n",
        "    parser.add_argument('--job-dir',  # handled automatically by AI Platform\n",
        "                        help='GCS location to write checkpoints and export ' \\\n",
        "                             'models')\n",
        "    parser.add_argument('--lr',  # Specified in the config file\n",
        "                        type=float,\n",
        "                        default=0.01,\n",
        "                        help='learning rate (default: 0.01)')\n",
        "    parser.add_argument('--momentum',  # Specified in the config file\n",
        "                        type=float,\n",
        "                        default=0.5,\n",
        "                        help='SGD momentum (default: 0.5)')\n",
        "    parser.add_argument('--hidden-layer-size',  # Specified in the config file\n",
        "                        type=int,\n",
        "                        default=8,\n",
        "                        help='hidden layer size')\n",
        "    args = parser.parse_args()\n",
        "    return args\n",
        "\n",
        "def train_model(args):\n",
        "    # Get the data\n",
        "    natality = pd.read_csv('https://storage.googleapis.com/ml-design-patterns/natality.csv')\n",
        "    natality = natality.dropna()\n",
        "    natality = shuffle(natality, random_state = 2)\n",
        "    natality.head()\n",
        "\n",
        "    natality_labels = natality['weight_pounds']\n",
        "    natality = natality.drop(columns=['weight_pounds'])\n",
        "\n",
        "\n",
        "    train_size = int(len(natality) * 0.8)\n",
        "    traindata_natality = natality[:train_size]\n",
        "    trainlabels_natality = natality_labels[:train_size]\n",
        "\n",
        "    testdata_natality = natality[train_size:]\n",
        "    testlabels_natality = natality_labels[train_size:]\n",
        "\n",
        "    # Normalize and convert to PT tensors\n",
        "    normalized_train = normalize(np.array(traindata_natality.values), axis=0)\n",
        "    normalized_test = normalize(np.array(testdata_natality.values), axis=0)\n",
        "\n",
        "    train_x = torch.Tensor(normalized_train)\n",
        "    train_y = torch.Tensor(np.array(trainlabels_natality))\n",
        "\n",
        "    test_x = torch.Tensor(normalized_test)\n",
        "    test_y = torch.Tensor(np.array(testlabels_natality))\n",
        "\n",
        "    # Define our data loaders\n",
        "    train_dataset = torch.utils.data.TensorDataset(train_x, train_y)\n",
        "    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True)\n",
        "\n",
        "    test_dataset = torch.utils.data.TensorDataset(test_x, test_y)\n",
        "    test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=128, shuffle=False)\n",
        "\n",
        "    # Define the model, while tuning the size of our hidden layer\n",
        "    model = nn.Sequential(nn.Linear(len(train_x[0]), args.hidden_layer_size),\n",
        "                          nn.ReLU(),\n",
        "                          nn.Linear(args.hidden_layer_size, 1))\n",
        "    criterion = nn.MSELoss()\n",
        "\n",
        "    # Tune hyperparameters in our optimizer\n",
        "    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)\n",
        "    epochs = 10\n",
        "    for e in range(epochs):\n",
        "        for batch_id, (data, label) in enumerate(train_dataloader):\n",
        "            optimizer.zero_grad()\n",
        "            y_pred = model(data)\n",
        "            label = label.view(-1,1)\n",
        "            loss = criterion(y_pred, label)\n",
        "            \n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "\n",
        "    val_mse = 0\n",
        "    num_batches = 0\n",
        "    # Evaluate accuracy on our test set\n",
        "    with torch.no_grad():\n",
        "        for i, (data, label) in enumerate(test_dataloader):\n",
        "            num_batches += 1\n",
        "            y_pred = model(data)\n",
        "            mse = criterion(y_pred, label.view(-1,1))\n",
        "            val_mse += mse.item()\n",
        "\n",
        "\n",
        "    avg_val_mse = (val_mse / num_batches)\n",
        "\n",
        "    # Report the metric we're optimizing for to AI Platform's HyperTune service\n",
        "    # In this example, we're mimizing loss on our test set\n",
        "    hpt = hypertune.HyperTune()\n",
        "    hpt.report_hyperparameter_tuning_metric(\n",
        "        hyperparameter_metric_tag='val_mse',\n",
        "        metric_value=avg_val_mse,\n",
        "        global_step=epochs        \n",
        "    )\n",
        "\n",
        "def main():\n",
        "    args = get_args()\n",
        "    print('in main', args)\n",
        "    train_model(args)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    main()"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting trainer/model.py\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jGT8Yv4u8xLP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a85369e5-6d20-41a3-ba48-8adb964c9725"
      },
      "source": [
        "%%writefile config.yaml\n",
        "trainingInput:\n",
        "  hyperparameters:\n",
        "    goal: MINIMIZE\n",
        "    maxTrials: 10\n",
        "    maxParallelTrials: 5\n",
        "    hyperparameterMetricTag: val_mse\n",
        "    enableTrialEarlyStopping: TRUE\n",
        "    params:\n",
        "    - parameterName: lr\n",
        "      type: DOUBLE\n",
        "      minValue: 0.0001\n",
        "      maxValue: 0.1\n",
        "      scaleType: UNIT_LINEAR_SCALE\n",
        "    - parameterName: momentum\n",
        "      type: DOUBLE\n",
        "      minValue: 0.0\n",
        "      maxValue: 1.0\n",
        "      scaleType: UNIT_LINEAR_SCALE\n",
        "    - parameterName: hidden-layer-size\n",
        "      type: INTEGER\n",
        "      minValue: 8\n",
        "      maxValue: 32\n",
        "      scaleType: UNIT_LINEAR_SCALE"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting config.yaml\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cl9OUzhtEGs4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "MAIN_TRAINER_MODULE = \"trainer.model\"\n",
        "TRAIN_DIR = os.getcwd() + '/trainer'\n",
        "JOB_DIR = BUCKET_URL + '/output'\n",
        "REGION = \"us-central1\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fhI4-4PHEC9k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create a unique job name (run this each time you submit a job)\n",
        "timestamp = str(datetime.datetime.now().time())\n",
        "JOB_NAME = 'caip_training_' + str(int(time.time()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VwAQAy_Dfm_2",
        "colab_type": "text"
      },
      "source": [
        "The command below will submit your training job to AI Platform. To view the logs, and the results of each HyperTune trial visit your Cloud console."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2WMfGk63Easm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "2a65ce59-6745-4ea4-f0f7-d0dfab32abf0"
      },
      "source": [
        "# Configure and submit the training job\n",
        "!gcloud ai-platform jobs submit training $JOB_NAME \\\n",
        "        --scale-tier basic \\\n",
        "        --package-path $TRAIN_DIR \\\n",
        "        --module-name $MAIN_TRAINER_MODULE \\\n",
        "        --job-dir $JOB_DIR \\\n",
        "        --region $REGION \\\n",
        "        --runtime-version 2.1 \\\n",
        "        --python-version 3.7 \\\n",
        "        --config config.yaml"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Job [caip_training_1589925625] submitted successfully.\n",
            "Your job is still active. You may view the status of your job with the command\n",
            "\n",
            "  $ gcloud ai-platform jobs describe caip_training_1589925625\n",
            "\n",
            "or continue streaming the logs with the command\n",
            "\n",
            "  $ gcloud ai-platform jobs stream-logs caip_training_1589925625\n",
            "jobId: caip_training_1589925625\n",
            "state: QUEUED\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQASUFOj2izH",
        "colab_type": "text"
      },
      "source": [
        "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
      ]
    }
  ]
}